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ABSTRACT 

A nonlinear cancellation technique for nonlinear voiccband 
channels is investigated. The technique uses a fast orthogonal 
search (FOS) algorithm to construct the cancellation filter by 
choosing terms from a large selection of candidates that have 
been selected after a theoretical investigation of the voiceband 
channel. This way it is possible to make a considerable reduc- 
tion in the number of terms needed for the nonlinear cancella- 
tion filter (NLCF). The performance of the technique is tested 
by simulations of nonlinear voiceband communication systems. 
The results demonstrate that the NLCF can be considered a 
practical alternative for improvement of modern modem re- 
ceivers 



1. INTRODUCTION 

It is well know that distortion in voiceband channels can have 
nonUnearities. Usually, the nonlinearities are caused by nonlin- 
ear circuits elements in telephone networks and have the forms 
of saturation. This source of nonlinearity is memory less but 
in combination with linear filtering in the system, the overall 
response is nonlinear with memory, and, thus, nonlinear inter- 
symbol interference (ISI) is induced. This ISI distortion has 
little or no effect on lower bit rate modems (2400 - 14.400 bps), 
but for higher bit rates (e.g. 28.800 bps and 33.600 bps) it can 
have an effect on the error rate. To be able to maintain these 
high bit rates for greater number of connections, than todays 
modems are cable of, an efficient method is needed to remove 
the nonlinear ISI. The objective of this paper is to cancel the 
nonlinear ISI with an algorithm which can be included in cur- 
rent modem designs without modifying the architecture of the 
receivers. 

There are many possible strategies to combat nonlin- 
ear ISI. Most of them use either equalization or cancella- 
tion [1], |2j, [3], (8|. Biglieri et al. [lj proposed two cancellation 
techniques. One can be included in present-day modems, and 
the other requires a change in the receiver architecture. Ser- 
faty et al. [8| also proposed cancellation techniques that can 
be used in existing modems. The drawback of all these tech- 
niques is that they require an enormous number of nonlinear 
terms for both a moderate length of memory and degree of 
nonlinearity. Also, most of them use orthogonalization in ev- 
ery adapting step which makes the computational burden too 
heavy for present-day modems. 

Here, a cancellation algorithm which uses Fast Orthogonal 
Search (FOS) |4), [5] to construct a truncated Volterra filter 
during start-up is investigated. The FOS selects which terms 
in the Volterra series should be used for the filter. The terms 
are selected from a predetermined set of candidates that have 
been chosen by examination of the theoretical response of the 
communication system which is modeled as a linear nonlinear 
linear (LNL) system. This allows us to reduce the number of 
possible candidate terms and make a considerable saving in 
computational cost. The FOS also finds the initial values of 
the coefficients of the filter. This way, the structure of the filter 
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is designed for the channel where it is used to compensate for 
distortion. This is important since the channel and the non- 
linear distortion vary from one connection to another. After 
the filter has been identified by the FOS, then it is possible 
to switch to a simple Least Mean Square (LMS) algorithm |7) 
to make the coefficients adaptive, so the nonlinear cancella- 
tion filter (NLCF) will be able to follow any slow changes in 
the channel distortion during one connection. The computa- 
tional cost of the algorithm is highest during start-up when the 
FOS is selecting terms, but this cost is within reasonable lim- 
its. After the start up phase, the computational burden drops, 
how much depends on the adaptive scheme used. This is very 
convenient because during the start-up phase most modems 
have some computational power to spare, since they do not 
use coded signals, such as Trellis code, until after the start-up 
phase. 

The paper is organized in the following way. First, a model 
of a nonlinear voiceband communication system is introduced . 
Section 3 gives a short description of the FOS algorithm. In 
Section 4 the NLCF is proposed and simulation results are 
discussed in Section 5. Finally, concluding remarks are given 
in Section 6. 

2. MODEL OF A NONLINEAR VOICEBAND 
COMMUNICATION SYSTEM 

The model used in this paper simulates a modem transmit- 
ter, a nonlinear voiceband channel and a modem receiver. 
The model is shown in Figure 1, the following description 
refers to that picture.. Box 1 shows the transmitter part 
of the modem. The data signal A n is complexed valued, 
A n = a n + jbny where Or and bn are two independent uni- 
formly distributed random variables which can take on the 
discrete levels { -31, -29, . . . , -1, 1, . . . , 29, 31} for QAM1024 
modulation (the number of points in the signal constellation 
set for V.34 and V.34+ are 960 and 1664 respectively). The 
data symbol rate was set to be 2400 symbol /sec. The data 
were upsampled by four to make room for the modulation. 
The LP1 is a pulse shaping FIR filter with 64 taps. The car- 
rier frequency was f c = 1700Hz. The second box in Figure 
1 models the nonlinear voiceband channel. Note that before 
the nonlinear distortion is added, the sample rate is doubled 
to avoid the occurrence of aliasing. LF2 and LP3 are 16 tap 
FIR filters used in the upsampling and downsampling proce- 
dure. Only third order nonlinear distortion is included. The 
coefficient 0 controls the third order harmonic content. The 
nonlinear distortion is subtracted form the original signal to 
simulate saturation. After the nonlinear distortion has been 
included, the sample rate is lowered by a factor two. The lin- 
ear part of the channel is implemented as a FTR filter. Box 3 
in Figure 1 shows the receiver part, of the modem. First the 
signal is demodulated, then it is sent through a matched filter 
LP4, identical to LP1. Thereafter, the signal is downsampled 
and fed into a \ fractional spaced linear equalizer (LE). From 
the above it can be seen that the system at hand is an LNL 
system. We shall now derive the response of such a system to 
a QAM modulated signal. For our application it is convenient 
to describe the QAM signal as 

x(t) = co${u^t)^2 n a n g(t - nT) 
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+sin(uMjt)Y^ n b n g(t — nT) 



(1) 



where u»o is the angular carrier frequency, ^ is the symbol rate 
and g(t) is the transmitter filter (LP1). The overall response 
of the LNL system in Figure 1 is 

y(t) = cos{u> 0 t) [£ k a k - d Hlt - kT) 

Ei ^2 m ak-dai^ d a m - d Gi(t -kT,t- IT, t - mT) 
Ei Em ^-d^dbm-dGaft - *T, t - -n\i^mT))-- 
[E fc ^-^(* " 

+^E* Ei L 6 *-*^-^^* - *T, t - /T, t - mT) 

Ei Em t>k-dOi- d a m --dGi(t - tr, t - /r, * - mT)j 

coa(3a;o«) [i^Et E; 53 m 
<2 5 (t - kT,t ~ IT, t - mT) 

+/*E* Ei E m o*-d*i-rf6*-4i<?.(t — A:T, < — IT, t - mT)] 

+*tn(3wof) [0E* Ei Em^-^-" 6 ^ dX 
C?7(i-Jfcr,t-/r,i-mT) 

+£E fc E t E m *fc-dai-d«m-dGfa(t - kT, t - IT, t - mT)] 

+«(«) (2) 

where ll(t) is the overall linear response of the system, 
Gi (ti , t 2 , * 3 ), . . . * C?6(ji , <2 , £3) define the cubic response of the 
system, and n(t) is Gaussian noise. The terms that are mod- 
ulated with angular frequency 3cuq are not expected to con- 
tribute much to the ISI because of lowpass filtering in the 
telephone networks. It can then be seen that in the baseband, 
the ISI can be. expected to have the form, . 

+£E fc Ei E m fl*-da/- rf a m . d G, (t - kT, t-lT t t- mT) - 
+/3E* Ei Em^-^'-^-^ 2 ^ - *T,i - iT y t - mT) 

+#£* Ei E^*-^-^-^3(( -kT,t - IT, t-mT) 
+/*E* Ei Em^-^-^m-d^lt - fcT, t - IT, t - mT) 

where d is the overall delay of the system. Ideally, the NLCF 
should be able to cancel the errors given in (3) and (4). In the 
next section, the FOS algorithm, which is a fundamental part 
of the proposed NLCF, will be described. 

3. THE FAST ORTHOGONAL SEARCH 

This section contains a short review of the FOS algorithm. 
This algorithm was proposed by Korenberg |4), |5) and can 
be used in identification of nonlinear systems with truncated' 
Volterra series- In the next section, it will be shown how the ■ 
FOS can l>e used as a part of an algorithm that effectively 
determines an NLCF. The algorithm enables one to pick out- 
the most effective terms in a Volterra series (effective in the : 
sense that these terms contribute more than other terms to the 
reduction of the mean squared error (MSE)), and thus enable 
us to constructed a compact model of the system. 

A discrete time Volterra series, with memory of length R 
and nonlinearity of degree rj t has the form 



y„(n) = fco-f ^ ki{ji)x(n - ji) 



Ji=0 



R— 1 fl-I 



+ 2^ »Ja) x ( n ~ 3i )*( n - 3i) H 

R~\ R-\ 

+ ^2 • ^2 h ^Uy*"- Jrt)x(n- ji)---x(n-j„)(5) 

j j =0 jr,=0 



w,here x(n) and y v (fi) are the input and output signal, respec- 
tively, and fco, . . . , k v are the Volterra kernels. The Volterra 
series can be written in a more convenient way as 



(6) 



where, M is the total number of terms in the truncated 
Volterra scries and o m presents the Volterra kernels. The goal 
of the identification is to select a new basis, for the filter which 
is a subspace of the space spanned by p m , m = 0, . . . , M - 1, 
based on the mean squared error criterion between t/«(n) and 
the desired output and thus reduce the number of coefficients 
in (6). This is done by constructing a orthogonal basis W m 
from pm such 



y v (n) ~ y^pmWm(n). 



(7) 



It can now be shown (6 J that the MSE between y v (ri) and a 
desired output y(n) is : 



(8) 



The FOS selects, at each orthogonalization ; step of p m , a 
new term, for the filter, which has the highest value of ^If^, 
among all the possible caiididat.es. This procedure is repeated 
until some predetermined maximum number of terms have 
been selected or until the MSE has gone below some prede- 
termined threshold. The FOS uses "fast" Cholesky decom- 
position to orthogonalize p m , and does not require a explicit 
calculation of <?* and Wi . 

4, THE NONLINEAJt CANCELLATION FILTER 

Here a cancellation scheme rather than equalization is chosen 
in order to prevent the need to find an inverse model for the 
nonlinear channel. The reason being that the inverse (if it 
exists) can have nonlinearities of higher order than the origi- 
nal model j3j. Here the structure for the NLCF is similar to 
the structure proposed by Biglieri et al. This structure is 
shown in Figure 2 where the NLCF is cascaded to the out- 
put of the equalizer and uses decision feedback (DFNLCF). 
Figure 2 shows the cancellation filter for the real part of the 
data signal, a identical cancellation filter is used for the imag- 
inary part. The main difference between our -NLCF and other 
filters previously- introduced in the literature is that here the 
inner-structure is not defined beforehand. Rather, the FOS 
algorithm is used to design the structure of the filter for every 
connection. Thus, a filter, can be designed in which all of its 
terms play an active role in reducing the lylSE at the output 
and prevent the problem of wasting computational power by 
including terms that have little or no effect on the systems 
; performance. ; t ; . 

The filter tries to model the, ISL Then the filter uses the 
model to predict the TSI and subtracts the prediction from the 
signal that is to be corrected. The DFNLCF uses precursor 
and postcursor information when predicting the ISI* therefore 
ti delay line is included at the output of the LE to make the sys- 
tem causal. Tentative decisions are used for postcursor sym- 
bols. The number of the linear and third order terms in (5) 

are ^ + ^ ~ * ) +7?* 'Wei see that the number of terms can 

be enormous for moderate length of memory. For example, 
the number of terms' for memory length 3 and 5 (thus in our 
, case R is 6 and 10 since we have two inputs for the Volterra 
filter) are 62 and 230 respectively. We would thus need 124 
and 460 terms, respectively, since we use two filters. The linear 
terms are included in the cancellation filter so it will \w. able 
to correct for linear ISI that are still remaining in the signal 
since the operation of the linear equalizer is affected by the 
nonlinearity in the channel. 

We propose a two step procedure to reduce the number of 
terms. First we use the results from (3) and (4). By using only 
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the terms in (3) and (4) the total number of terms in the pre- 
vious example are reduced from 124 and 460 terms to 68 and 
240 terms, respectively. By doing this we make a considerable 
saving in the computational power needed for the FOS which 
reduces the number of terms much further. 

The FOS algorithm is not adaptive. However, it is necessary 
to make the NLCF adaptive to compensate for slow variations 
in the channels response. There are many -possible ways to 
accomplish this |7J. It is important to note that the FOS 
finds the initial values of the filter coefficients and the adaptive 
algorithm has only to be able to follow the slow changes in the 
channel. This allows the use of simple algorithms like LMS or 
normalized LMS. 

5. SIMULATION RESULTS 

The Q AM1024 system described in Section 2 was used for sim- 
ulation. In all the simulation the linear equalizer was trained 
on the first 2 * 20000 symbols and then its adaption was 
stopped, the nonlinear cancellation filter was then identified 
with FOS, where 2 * 2000 training symbols where used. In the 
first simulation the SNR was held constant at about 41dB for 
all runs. The third order nonlinear distortion was then varied 
from 29.7dB to 37.3dB. The system was simulated until the 
uuuiber of errors had reached 500. In the first simulation we 
compared three NLCF. Filter cue used all the possible linear 
and third order terms in a Volterra filter with memory length 

3 and delay 1, i.e., it had 2 * 62 = 124 terms. The memory 
length of filter two was also 3 and the delay was 1, but it used 
only the terms shown in (3) and (4) and linear terms, and 
had 2 * 34 = 68 terms. The coefficients of filter one and two 
were identified using the FOS algorithm without any term se- 
lection. Filter three had memory length 5 and delay 2. The 
terms of filter three were chosen from the terms in (3), (4) 
and linear terms, i.e., it had 2*120=240 candidate terms (a 
reduction from 2*230= 460 candidate terms). Then 2*34=68 
terms were selected from the candidate terms using the FOS 
algorithm. Figure 3 shows the results. The first line shows the 

■ probability of error for system with no NLCF. Line labeled 2 
shows the performance of filter one, line labeled 3 shows the 
performance of filter two and line labeled 4 shows the perfor- 
mance of filter three. Figure 3 shows that filters one and two 
perform identically, which supports' the approximations made 
in (3); and (4). Filter three achieves considerable better per- 
formance than filter two, with the same number of coefficients. 
This is due to the fact that filter three is able to include terms 
with grater lag than filter two. -When the third harmonic dis- 
tortion are 32.5dB and 34.5dB filter (three reduces- the effect 
of the nonlinear distortion by about 2.4dB and, 2ildB, respec- 
tively, While filter two reduces, the nonlinear distortion by only 
about 1 . 7dB and 1 .5dB, respectively. < 

In the second simulation third order : nonlinear distort iou . ■* 
was held constant at about 32.5dB and 'the 'SNR' 'Was-" varied 
from 30db to 73dB. Simulations were niade for filter two, three- 
and four. Here filter four has memory length 3 and delay 1, 
and number of coefficients' were, reduced from -2 * 34 = 68 to 
2*6 = 12 terras by FOS. Figure 4 shows' the 1 results. The line 
labeled 1 shows the probability of erf of for system with no 
NLCF, line labeled 2 shows the performance of filter four, line 
labeled 3 shows the performance of filter two and line labeled 

4 shows the probability of error for filter thre^.' Finally, line 
labeled 5 shows the performance of a system with no nonlinear 
distortion. We see from Figure 4 that filter three dose not 
show a better performance than filters two and four until the 
nonlinear distortion is 4dB higher than, the ,Ganssiari noise 
in the system. It is likely that the reason for this is that 
the nonlinear ISI terms have less power as the lag grows and 
are therefore buried in 1 noise when the SNR and -the nonlinear 
distortion are similar in power. Which leads to a difficulty 
in estimating the value of those terms. This makes it hard or 
even impossible to completely cancel the effect of the nonlinear 
distortion. 

In the last simulation only a system with filter three and 
four were simulated, but the number of terms in the filter one 
were varied from 2*1 = 2 to 2*73 — 146 and for filter four 
from 2*1 = 2 to 2* 33 = 66, in steps of size two terms. The 



SNR and third order nonlinear distortion were held constant 
at about 41dB and 32.5dB respectively. We see from Figure 5 
that there is no gain in including more than 2 * 41 = 82 terms 
in filter three and if more terms are added then we get worse 
performance. Similar behavior is seen for filter four, there 
nothing is gained hy including more than 2 * 9 = 18 terms. 
The reason for this is probably that the nonlinear error does 
not lie in the space spanned by the rest of the terms. These 
term will therefore only try to model the noise in the training 
data. The simulation results suggest that only about 15% of 
the available candidate terms play an active role in reducing 
the MSE. 

6. CONCLUSION 

An. algorithm for cancellation of nonlinear intersymbol inter- | 
ference (ISI) in voiceband communication systems has been in- 
vestigated. The algorithm uses fast orthogonal search (FOS) to 
construct a nonlinear cancellation filter (NLCF) at the begin- j 
ning of the connection, from a predetermined set of candidate , ; 
terms that have been selected by theoretical investigation of 
the channels response. Then the algorithm can be made adap- 
tive by the use of the LMS algorithm or any other adaptive 1 
algorithm, since the resulting filter is linear in its coefficients. 
The algorithm is easy to implement and does not rely heav- 
ily on computational resources when implemented with lookup 
tables. Such implementation is possible when the same input j 
data series is used for the identification with FOS. The al- 
gorithm can be looked at as an extension to modern modem 
receivers and it requires no change in the receiver architecture. i 
The NLCFs which were constructed with term selection out- j 
perform the other NLCF tested in the simulation. In voice- | 
band communication channels the expected SNR can ranges'! 
from 34dB to 42dB. The simulation showed that the algorithm *^ J_ 
can perform well in this range and give a considerable gain in ■ • c 
error probability. In the experiments it„was shown that it is.;, j ^ 
possible to reduce the number of terms in the Volterra filter " 7 z 
with a given memory by more than 85%^ without having effect ^ ^ 
on its performance. ^ ; £ 
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Figure 1. Blockdiagram of the communication system used in the simulations. 
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Volterra filter 



Figure 2. Blockdiagram of the nonlinear 
cancellation filter. 




Signal to thrid harmonic distortion ratio (dB) 
Figure 3. Results from the first simulation, 
see text. 




Signal to thrid harmonic distortion ratio (dB) 
Figure 4. Results from the second simulation, 
sec text. 




Number of terms 
Figure 5. Results from the third simulation, 
see text: 
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